feat(run-environment): support CircleCI for GitHub repositories by fargito · Pull Request #482 · CodSpeedHQ/codspeed

fargito · 2026-07-28T14:30:39Z

Add a CircleCI run environment, for repositories hosted on GitHub.

This replaces the former stack (#473 → #482) with a single PR against main.
Two commits: the CircleCI provider, and the metadata payload cleanup that came
with it.

The build runs on CircleCI while commits and pull requests live on GitHub, so
the run is reported against the GitHub commit and pull request. Detection is
CIRCLECI=true, and the repository comes from CIRCLE_REPOSITORY_URL.

CircleCI also builds Bitbucket and GitLab repositories, so the remote's domain
is checked and anything but github.com is rejected with an error naming the
domain found. That domain is the only signal available at runtime:
pipeline.project.type is a pipeline value, interpolated when the config is
compiled, so it never reaches the job as an environment variable.

Uploads authenticate with an OIDC token

The job mints its own token, so no CODSPEED_TOKEN secret is needed. The token
CircleCI exposes in CIRCLE_OIDC_TOKEN and CIRCLE_OIDC_TOKEN_V2 cannot be
used: its audience is fixed to the id of the CircleCI organization, while
CodSpeed requires its own. Only the circleci CLI can request a custom audience
— CircleCI publishes no endpoint for it, unlike the
ACTIONS_ID_TOKEN_REQUEST_URL GitHub Actions provides — so the runner shells
out to circleci run oidc get, once per upload, as a token expires an hour
after it is minted.

Whether a job can mint at all is settled by minting a token and throwing it
away, before the benchmarks run. A CLI can be installed and still predate
run oidc get, so probing for the binary would pass and the mint would fail
once the benchmarks had run for nothing.

Two cases keep needing a static token, and both are refused upfront: pull
requests opened from a fork, whose token names the fork, and jobs that cannot
mint at all. A third is only visible when the upload is refused: a pipeline
triggered by a custom webhook gets a token naming no repository.

The error messages link integrations/ci/circleci/configuration, anchors
#authentication and #oidc-recommended. That page does not exist yet, and
owes the three cases above.

Worth a look

No commit hash override. CircleCI checks out the associated commit rather
than a synthetic merge ref, so reading git HEAD already yields
CIRCLE_SHA1.
runId is CIRCLE_WORKFLOW_ID, shared by every job and every parallel
container of a workflow. The per-job CIRCLE_WORKFLOW_JOB_ID would split one
workflow into unrelated runs. runPartId is
{CIRCLE_JOB}-{CIRCLE_NODE_INDEX}, to stay unique along both fan-out axes.
ghData and glData are gone from the metadata, which makes it
version 11. Both carried a run id and a job name that runPart has carried
for every provider since version 6, so the payload named the same job twice,
and only ever for GitHub Actions and GitLab CI — every other provider filled
both with None to say it was neither of them. The backend accepts both
shapes: it reads either field only when runPart is missing.
baseRef and sender are left empty. CircleCI exposes neither a
base-branch variable nor the id the repository provider gives the user who
triggered a build. The backend resolves both from the repository provider: the
base branch from the pull request, and the run's author from its commit.

Backend support is not released yet, so a real CircleCI job cannot be verified
end to end.

Closes COD-2995 Closes COD-3257

greptile-apps · 2026-07-28T14:33:09Z

Greptile Summary

The PR adds CircleCI support for GitHub-hosted repositories and updates the upload metadata contract.

Detects CircleCI jobs and derives repository, pull-request, workflow, job, and parallel-container metadata.
Preflights CircleCI OIDC by minting a token before benchmarks, then mints a fresh token for each upload.
Removes redundant GitHub/GitLab metadata fields and advances the upload metadata version to 11.
Adds CircleCI-specific logging and authentication guidance.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the earlier compatibility-probe issue is fixed by exercising the actual OIDC mint command before benchmark execution.

Important Files Changed

Filename	Overview
src/run_environment/circleci/oidc.rs	Implements custom-audience token minting and preserves CircleCI CLI failures without exposing successful token output.
src/run_environment/circleci/provider.rs	Implements CircleCI detection, GitHub repository validation, run metadata, parallel run parts, and preflight/per-upload OIDC handling.
src/run_environment/interfaces.rs	Adds the CircleCI environment variant and removes redundant provider-specific metadata fields.
src/upload/interfaces.rs	Advances the upload metadata schema version to 11.
src/upload/uploader.rs	Adds CircleCI-specific authentication guidance for rejected uploads.

Sequence Diagram

sequenceDiagram
    participant Job as CircleCI job
    participant Runner as CodSpeed runner
    participant CLI as CircleCI CLI
    participant Backend as CodSpeed backend
    Job->>Runner: Start benchmark command
    Runner->>Runner: Detect CircleCI and build provider
    Runner->>CLI: circleci run oidc get
    CLI-->>Runner: Preflight token
    Runner->>Runner: Execute benchmarks
    loop Each upload
        Runner->>CLI: Mint fresh OIDC token
        CLI-->>Runner: Upload token
        Runner->>Backend: Upload metadata and profile
    end

_{Reviews (3): Last reviewed commit: "refactor(upload): drop ghData and glData..." | Re-trigger Greptile}

codspeed-hq · 2026-07-28T14:37:09Z

Merging this PR will not alter performance

⚠️

Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 17 untouched benchmarks

_{Comparing feat/circleci-oidc-upload-auth (916a1bd) with main (6eac757)}

fargito · 2026-07-28T15:09:48Z

@greptileai review

The build runs on CircleCI while commits and pull requests live on GitHub, so a run is reported against the GitHub commit and pull request. Detection is `CIRCLECI=true`, and the repository comes from `CIRCLE_REPOSITORY_URL`. CircleCI also builds Bitbucket and GitLab repositories, so the remote's domain is checked and anything but `github.com` is refused with an error naming the domain found. That domain is the only signal available at runtime: `pipeline.project.type` is a pipeline value, interpolated when the config is compiled, so it never reaches the job as an environment variable. `runId` is `CIRCLE_WORKFLOW_ID`, shared by every job and every parallel container of a workflow; the per-job `CIRCLE_WORKFLOW_JOB_ID` would split one workflow into unrelated runs. `runPartId` is `{CIRCLE_JOB}-{CIRCLE_NODE_INDEX}`, to stay unique along both fan-out axes. No commit hash override is needed, as CircleCI checks out the associated commit rather than a synthetic merge ref. Uploads authenticate with an OIDC token the job mints for itself, so they need no `CODSPEED_TOKEN` secret. The token CircleCI exposes in `CIRCLE_OIDC_TOKEN` and `CIRCLE_OIDC_TOKEN_V2` cannot be used: its audience is fixed to the id of the CircleCI organization, while CodSpeed requires its own. Only the `circleci` CLI can request a custom audience — CircleCI publishes no endpoint for it, unlike the `ACTIONS_ID_TOKEN_REQUEST_URL` GitHub Actions provides — so the runner shells out to `circleci run oidc get`, once per upload, as a token expires an hour after it is minted. Whether a job can mint at all is settled by minting a token and throwing it away, before the benchmarks run: a CLI can be installed and still predate `run oidc get`, so probing for the binary would pass and the mint would fail once the benchmarks had run for nothing. A static `CODSPEED_TOKEN` is used as is when one is set, and stays required for a pull request opened from a fork, whose token names the fork. `baseRef` and `sender` are left empty: CircleCI exposes neither a base-branch variable nor the id the repository provider gives the user who triggered a build. Closes COD-2995 Closes COD-3257

…adata Both carried a run id and a job name that `runPart` already carries for every provider, so the payload named the same job twice, and only ever for GitHub Actions and GitLab CI. Every other provider filled both fields with `None` to say it was neither of them. GitHub Actions and GitLab CI keep the two values as plain fields read when the provider is built, the way CircleCI holds the ones its own run part needs. The payload no longer has the shape version 10 describes, so it becomes version 11. The backend accepts both: it reads either field only when `runPart` is missing, which no version since 6 has been. Refs COD-3257 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

fargito · 2026-07-30T13:43:28Z

@greptileai review

adriencaccia · 2026-07-30T15:14:53Z

+        if let Some(group_event) = get_group_event(record) {
+            match group_event {
+                GroupEvent::Start(name) | GroupEvent::StartOpened(name) => {
+                    println!("{}", style(name).cyan().bold());
+                }
+                GroupEvent::End => {}
+            }
+            return;
+        }
+
+        if get_json_event(record).is_some() {
+            return;
+        }
+
+        if let Some(announcement) = get_announcement_event(record) {
+            println!("{}", style(announcement).green());
+            return;
+        }
+
+        if level > self.log_level {
+            return;
+        }
+
+        match level {
+            Level::Error => {
+                println!("{}", style(message).red());
+            }
+            Level::Warn => {
+                println!("{}", style(message).yellow());
+            }
+            Level::Info => {
+                println!("{message}");
+            }
+            Level::Debug => {
+                println!("{}", style(message).cyan());
+            }
+            Level::Trace => {
+                println!("{}", style(message).magenta());
+            }
+        }
+    }


Can't we do better for logging?

fargito mentioned this pull request Jul 28, 2026

feat(run-environment): add CircleCI provider for GitHub repositories #473

Closed

greptile-apps Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread src/run_environment/circleci/oidc.rs Outdated

fargito commented Jul 28, 2026

View reviewed changes

Comment thread src/upload/uploader.rs

fargito force-pushed the feat/circleci-oidc-upload-auth branch from 12d9f72 to 600b868 Compare July 28, 2026 15:39

fargito commented Jul 28, 2026

View reviewed changes

Comment thread src/run_environment/circleci/provider.rs

fargito requested a review from adriencaccia July 28, 2026 16:05

fargito changed the title ~~feat(run-environment): mint CircleCI OIDC tokens for upload auth~~ feat(run-environment): support CircleCI for GitHub repositories Jul 30, 2026

fargito changed the base branch from feat/circleci-run-environment to main July 30, 2026 13:38

fargito force-pushed the feat/circleci-oidc-upload-auth branch from e2e325d to dfa68d0 Compare July 30, 2026 13:38

fargito and others added 2 commits July 30, 2026 15:43

fargito force-pushed the feat/circleci-oidc-upload-auth branch from dfa68d0 to 21b4ee9 Compare July 30, 2026 13:45

chore: Release codspeed-runner version 5.0.2-alpha.1 🎉

916a1bd

adriencaccia approved these changes Jul 30, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(run-environment): support CircleCI for GitHub repositories - #482

feat(run-environment): support CircleCI for GitHub repositories#482
fargito wants to merge 3 commits into
mainfrom
feat/circleci-oidc-upload-auth

fargito commented Jul 28, 2026 •

edited

Loading

Uh oh!

Uh oh!

greptile-apps Bot commented Jul 28, 2026 •

edited

Loading

Important Files Changed

Sequence Diagram

Uh oh!

codspeed-hq Bot commented Jul 28, 2026 •

edited

Loading

Uh oh!

Uh oh!

fargito commented Jul 28, 2026

Uh oh!

Uh oh!

fargito commented Jul 30, 2026

Uh oh!

adriencaccia Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

fargito commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uploads authenticate with an OIDC token

Worth a look

Uh oh!

Uh oh!

greptile-apps Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Sequence Diagram

Uh oh!

codspeed-hq Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

Uh oh!

fargito commented Jul 28, 2026

Uh oh!

Uh oh!

fargito commented Jul 30, 2026

Uh oh!

adriencaccia Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

fargito commented Jul 28, 2026 •

edited

Loading

greptile-apps Bot commented Jul 28, 2026 •

edited

Loading

codspeed-hq Bot commented Jul 28, 2026 •

edited

Loading